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ABSTRACT 



Much of the literature on random-digit-dial (RDD) response 
rates focuses on the benefits and drawbacks of various screening and sampling 
procedures. The common assumption is that enumeration is more invasive, 
leading to lower response rates, but there are concerns about population 
coverage and self -selection with other methods. When the target population is 
found in a subset of households, a screen-out question may be used to 
eliminate ineligible households prior to employing sampling methods. Using 
approximately 10,000 telephone numbers from the National Household Education 
Survey (NHES) sampling frame, this experiment examined the impact on 
screening response rates of: full enumeration of all households (no 
screen-out) versus a screen-out question, and mailing an advance letter. An 
RDD sample was divided into four quarter samples: (1) screen-out and letter,* 

(2) screen-out and no letter,* (3) no screen-out and letter; and (4) no 
screen-out and no letter. Response rates were significantly higher in the 
screen-out condition. The advance mailing increased cooperation in the no 
screen-out condition, but not in the screen-out condition. The no screen-out 
condition consumed substantially more resources than a screen-out sample of 
the same size, but also provided more data for each completed case. 
Implications of this experiment for the design of future surveys are 
discussed. Two appendixes present the NHES field test letter and the advance 
letter. (Contains 7 tables, 1 exhibit, and 24 references.) (Author/SLD) 



**************++++++++++++*++++**★*****★★**********+++*+******++++++*+++******** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document . * 

******************************************************************************** 




TM 027 974 



NATIONAL CENTER FOR EDUCATION STATISTICS 



Technical Report December 1997 


National Household Education Survey 


An Experiment in 




Random-Digit-Dial 




Screening 




National 


U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 


Household 


> CENTER (ERIC) 

(a This document has been reproduced as 
received from the person or organization 
originating it. 


Education 


□ Minor changes have been made to 
improve reproduction quality. 


Survey 


• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



U.S. Department of Education 

Office of Educational Research and Improvement NCES 98-255 



ERIC 



2 



BEST COPY AVAILABLE 



NATIONAL CENTER FOR EDUCATION STATISTICS 



Technical Report 



December 1997 



National Household Education Survey 



An Experiment in 

Random-Digit-Dial 

Screening 




National 

Household 

Education 

Survey 



J. Michael Brick 
Mary Collins 
Westat, Inc. 

Kathryn Chandler 

National Center for Education Statistics 



U.S. Department of Education 

Office of Educational Research and Improvement NCES 98-255 




9 



U.S. Department of Education 

Richard W. Riley 
Secretary 

Office of Educational Research and Improvement 

Ricky T. Takai 

Acting Assistant Secretary 

National Center for Education Statistics 

Pascal D. Forgione, Jr. 

Commissioner 

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, 
and reporting data related to education in the United States and other nations. It fulfills a congressional 
mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in 
the United States; conduct and publish reports and specialized analyses of the meaning and significance 
of such statistics; assist state and local education agencies in improving their statistical systems; and 
review and report on education activities in foreign countries. 

NCES activities are designed to address high priority education data needs; provide consistent, reliable, 
complete, and accurate indicators of education status and trends; and report timely, useful, and high 
quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, 
practitioners, data users, and the general public. 

We strive to make our products available in a variety of formats and in language that is appropriate to a 
variety of audiences. You, as our customer, are the best judge of our success in communicating 
information effectively. If you have any comments or suggestions about this or any other NCES product or 
report, we would like to hear from you. Please direct your comments to: 

National Center for Education Statistics 
Office of Educational Research and Improvement 
U.S. Department of Education 
555 New Jersey Avenue NW 
Washington, DC 20208-5574 



December 1997 

The NCES World Wide Web Home Page is 
http://www.ed.gov/NCES/ 

Suggested Citation 

U.S. Department of Education. National Center for Education Statistics. An Experiment in Random-Digit- 
Dial Screening, NCES 98-255, by J. Michael Brick, Mary Collins, and Kathryn Chandler. Washington, DC: 
1997. 

Content Contact: 

Kathryn Chandler 
(202)219-1767 
(e-mail) NHES@ed.gov 
httpp;//www. ed.gov/NCES/NHES 

For free single copies of this publication, call the National Library of Education at 1-800-424-1616 or 
send a FAX request to (202) 219-1696. 



TABLE OF CONTENTS 



Section Page 

ABSTRACT 1 

Introduction 1 

Background of the National Household Education Survey 1 

Rationale for an Experiment in Screening 2 

Respondent Selection 3 

Mailings to Respondents 4 

Design of the Experiment 4 

Data Collection Procedures 6 

Outcome Measures : 8 

Findings 1 1 

Cooperation, Conversion, and Response Rates 1 1 

Eligibility Rates t. 16 

Resources 16 

Conclusions 17 

References 21 

Appendix A: NHES:96 Field Test Screener 23 

Appendix B: Advance Letter 4 1 





LIST OF TABLES 



1 . Number of telephone numbers, by screen-out question and 

advance letter 7 

2. Number of telephone numbers dialed, by final Screener response 

category and screen-out question 9 



3. Number of Screener, Screener refusal conversion, and extended 
interview cases, by cooperation rates, screen-out question, 
and advance letter 



4. Differences in cooperation rates, by screen-out question and 

advance letter 1 2 

5. Predicted Screener response rate, by screen-out question and advance 

letter 13 

6. Number of households and gross Screener response rate, by screen-out 

question and advance letter 1 5 

7. Administration time of Screeners, by experimental condition 

and household type 18 



LIST OF EXHIBITS 



1 . Definitions of Screener results for calculation of outcome 

measures 9 



3 

ERIC 



IV 



ABSTRACT 



Much of the literature on RDD response rates focuses on the benefits and drawbacks of various screening and 
sampling procedures. The common assumption is that enumeration is more invasive, leading to lower response 
rates, but there are concerns about population coverage and self-selection with other methods. When the target 
population is found in a subset of households, a screen-out question may be used to eliminate ineligible households 
prior to employing sampling methods. This experiment examined the impact on screening response rates of (1) full 
enumeration of all households (no screen-out) versus a screen-out question and (2) mailing an advance letter. An 
RDD sample was divided into four quarter-samples: screen-out and letter; screen-out and no letter; no screen-out 
and letter; no screen-out and no letter. Response rates were significantly higher in the screen-out condition. The 
advance mailing increased cooperation in the no screen-out condition, but not in the screen-out condition. The no 
screen-out condition consumed substantially more resources than a screen-out sample of the same size, but also 
provided more data for each completed case. Implications of this experiment for the design of future surveys are 
discussed. 



Introduction 

One of the challenges of random-digit-dial 
(RDD) surveys is to obtain the cooperation of 
households at the beginning of a household 
contact. Unlike list-sample surveys, in which 
sampled persons are asked for by name and are 
often connected to a sponsoring or cooperating 
institution, little is known about the telephone 
numbers selected for an RDD study, and 
respondents may perceive the call as a 
telemarketing attempt or an unwanted intrusion. 
A household screening response rate in the 1995 
National Household Education Survey 
(NHES:95) that was significantly lower than 
previous NHES administrations led to an interest 
in identifying the source of the loss in response 
and exploring survey designs that maximize 
response rates at the screening level. 

Because the NHES:95 was the first survey in this 
ongoing data collection system to use full 
enumeration of all household members in all 
sampled households, it was suspected that this 
approach was a likely factor in the decline of the 
screening response rate. A systematic 
experiment was developed and executed to 
examine the impact of the full enumeration 
approach on survey response. The experiment 
also included a test of an advance letter to 
households for which addresses could be 
obtained, based on the success of a nonresponse 
letter utilized in the NHES:95. 
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This report presents the design and results of this 
experiment in RDD screening. It begins with an 
overview of the NHES survey system, then 
proceeds through the design, data collection and 
analysis of the data from the experiment. 

Background of the National Household 
Education Survey 

The National Household Education Survey 
(NHES) is a data collection system of the 
National Center for Education Statistics (NCES), 
which has as its legislative mission the collection 
and publication of data on the condition of 
education in the Nation. The NHES is 
specifically designed to support this mission by 
providing information on those educational issues 
that are best addressed by contacting households 
rather than schools or other educational 
institutions. The NHES provides descriptive data 
on the educational activities of the U.S. 
population and offers policymakers, researchers, 
and educators a variety of statistics on the 
condition of education in the United States. 

The NHES is a telephone survey of the 
noninstitutionalized civilian population of the 
U.S. Households are selected for the survey 
using random digit dialing (RDD) methods, and 
data are collected using computer-assisted 
telephone interviewing (CATI) procedures. From 
45,000 to 64,000 households are screened for 
each administration, and individuals within 
households who meet predetermined criteria for 
populations of interest in the given survey 
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(usually defined by age or grade in school) are 
sampled for more detailed or extended 
interviews. The data are weighted to permit 
estimates of the entire population. The NHES 
survey for a given year typically consists of a set 
of screening questions (Screener), which collects 
household composition and demographic data, 
and two extended interviews on different 
education-related topical components. In order to 
assess data item reliability and inform future 
NHES surveys, each administration also includes 
a subsample of respondents for a reinterview. 

Throughout its history, the NHES has collected 
data in ways that permit estimates to be tracked 
across time. This includes repeating full topical 
components on a rotating basis in order to 
provide comparative data across survey years. In 
addition, each administration of the NHES has 
benefited from experiences with previous cycles, 
resulting in improvements to the survey 
procedures and content. Thus, while the survey 
affords the opportunity for tracking phenomena 
across time, it is also dynamic in addressing new 
issues and including conceptual and 

methodological refinements. 

A new design feature implemented in the 
NHES:96 is the collection of demographic and 
educational information on members of all 
screened households, rather than just those 
households potentially eligible for a topical 
component. In addition, this expanded screening 
feature included a brief set of questions on an 
issue of interest to education program 
administrators or policymakers. The total 
Screener sample size was sufficient to produce 
state estimates of household characteristics for 
the NHES:96. 

The NHES was conducted in 1991, 1993, 1995, 
and 1996. Topics addressed by the NHES:91 
were early childhood education and adult 
education. The NHES:93 collected information 
about school readiness and school safety and 
discipline. The 1991 components were repeated 
for the NHES:95, addressing early childhood 
program participation and adult education. Both 
components underwent substantial redesign to 
incorporate new issues and develop new 



measurement approaches. In the NHES:96, the 
topical components were parent/family 
involvement in education (PFI) and civic 
involvement (Cl). The NHES:96 expanded 
Screener included a set of questions on public 
library use. 

The experiment discussed in this report was 
conducted in preparation for the NHES:96 full 
scale survey. Although most of the issues raised 
are pertinent to RDD surveys in general, some of 
the specific features of the NHES:96 such as the 
Parent PFI/CI and Youth Cl interviews, and the 
public library questions included in the expanded 
screener are mentioned. Readers interested in 
more detail on the topics covered in the NHES:96 
can refer to Collins et al. (forthcoming). 

In addition to its topical components, the NHES 
system has also included a number of 
methodological investigations. These have 
resulted in technical reports and working papers 
covering diverse topics such as telephone 
undercoverage bias, proxy reporting, and 
sampling methods. This series of technical 
reports and working papers provides valuable 
information on ways of improving the NHES. 
The present research reflects a continuation of 
this concern for methodological issues by 
evaluating factors associated with response rates 
in RDD surveys and the resources associated with 
selected approaches. 

Rationale for an Experiment in 
Screening 

As noted above, the idea of conducting an 
experiment in the NHES on factors associated 
with Screener response rates was precipitated in 
part by a lower than expected screening response 
rate in the NHES:95. While the NHES:93 
response rate at the screening stage was 82 
percent (Brick et al. 1994), the NHES:95 
screening response rate was 73 percent (Collins et 
al. 1996). As mentioned above, the topics 
addressed in the surveys were different, but both 
survey administrations were related to 
educational issues and the topics covered were 
not expected to be associated with the screening 
response rates. 
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A key design difference between the 1993 and 
1995 surveys was the screening method. In the 
1993 survey, only the 30 percent of households 
with at least one person age 1 8 or younger or in 
12th grade or below were enumerated in the 
Screener. A screen-out question was asked in the 
Screener and households without any potentially 
eligible members were eliminated without 
enumerating all the household members. In the 
1995 survey, however, every household was 
potentially eligible for an extended interview and 
all the members of every household were 
enumerated in the Screener. It was suspected that 
this full enumeration of all households was a 
major factor in the difference in response rates. 

When it was clear that the Screener response rate 
in the NHES:95 was going to be much lower than 
planned, letters were mailed to nonresponding 
households, except those designated as "language 
problems." The addresses were obtained from a 
commercial service and fewer than half of the 
telephone numbers could be matched to valid 
addresses of households. Since addresses were 
not available for all telephone numbers, the letter 
was mailed only to those households for which 
addresses were obtained. The results of this 
NHES:95 nonresponse mailing showed that those 
who were mailed letters responded at a much 
higher rate than those who were not sent letters. 
This raised the question as to the potential 
effectiveness of an advance mailing to an RDD 
sample. 

The variation in response rates for RDD surveys 
of households is well documented, but not well 
understood. Some RDD surveys obtain response 
rates of 50 percent or less while others have 
response rates of 80 percent or more (Groves and 
Kahn 1979; Thomberry and Poe 1982; Friedrichs 
1987; Alexander et al. 1986; Salmon and 
Nicholls 1983; Hagan and Collier 1983). The 
variation in telephone survey response rates is 
associated with factors including the content of 
the survey, sponsorship, the length of the 
interview, and data collection procedures such as 
callback protocols and refusal conversion efforts 
(Groves and Lyberg 1988). However, it is 
difficult to estimate the relative effects of specific 
aspects of surveys because these aspects vary 



concurrently and there are no definitive 
experiments on the impact of these factors on 
response rates. As a result of this uncertainty, an 
experiment involving some factors that were 
feasible in the NHES was thought to have value. 
This research provides the opportunity to test the 
effects of certain experimental conditions while 
holding constant the survey content, timing, 
interviewer training, and calling protocol. 

Respondent Selection 

Much of the literature on response rates and 
screening methods in RDD surveys focuses 
primarily on procedures used to sample adult 
household members and the relative benefits and 
drawbacks of various procedures. This literature 
examines alternatives to a full enumeration 
approach like the Kish method of respondent 
selection, which is similar to the method used in 
the NHES:95. The Kish method involves the 
enumeration of all adult household members by 
age and sex at the start of the screener or 
interview (Kish 1949). Because of concerns 
about the intrusiveness of collecting such detailed 
household information early in an interview and 
the amount of time required to conduct the 
enumeration, there has been concern that the Kish 
method has a depressive effect on response rates 
(Lavrakas et al. 1993; Forsman 1993; Oldendick 
et al. 1988). A number of alternative procedures 
for selecting adult respondents within households 
have been developed and compared to the Kish 
method. 

The Troldahl-Carter method (Troldahl and Carter 
1964) involves ascertaining the number of adult 
household members and using an alternating 
procedure to select the oldest male, oldest female, 
youngest male, or youngest female. Another 
method that has been used increasingly in recent 
years is the "last birthday" (or sometimes "next 
birthday") method (Lavrakas et al. 1993). Under 
this procedure, the screener respondent is asked 
to identify the adult age 18 or older in the 
household who has most recently had a birthday 
(or will be the next to have a birthday). It is 
necessary to ascertain the total number of adults 
in the household for appropriate weighting. 
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Much of the research literature on RDD screening 
focuses on differences in rates of refusal in 
studies that manipulate the method of respondent 
selection (Oldendick et al. 1988; Salmon and 
Nicholls 1983; Hagan and Collier 1983; Czaja et 
al. 1982) or the placement of the enumeration 
within an instrument (Bercini and Massey 1979; 
Alexander et al. 1986). The results of these 
studies are somewhat mixed, and it is not clearly 
demonstrated that the alternatives to enumeration 
methods provide uniformly higher response rates 
(Czaja et al. 1982; Oldendick et al. 1988; 
O'Rourke and Blair 1983). In addition, these 
methods of respondent selection may also have 
an impact on other features of the sample such as 
population coverage and inappropriate 
respondent self-selection (Bryant 1975; Carr and 
Hertvik 1993; Hagan and Collier 1983; Lavrakas 
et al. 1993; Romuald and Haggard 1994). 

In general, this literature focuses almost 
exclusively on methods of sampling one random 
adult within households. The sampling of 
children as a unit of analysis is seldom, if ever, 
addressed. When children are sampled or there is 
the potential to sample more than one person in a 
household for a survey, as in the NHES, 
enumeration becomes an essential design 
element. In each of the four full-scale NHES 
collections conducted to date, households with 
children could contain one or more eligible 
members, and in two of the collections, adults 
were sampled based on data collected in the 
Screener. The enumeration of households 
members in these situations may be needed so 
that information required for sampling household 
members can be collected in an unambiguous 
way. Similarly, when it is likely that all, or 
nearly all, households include at least one person 
eligible for a study, the full enumeration of all 
households may be important. However, when 
the population of interest is found in a relatively 
small fraction of households, a preliminary 
screen-out question may be used prior to 
enumeration to eliminate those households 
without any eligible members. Such eligibility 
criteria can be based on age or on other 
characteristics that can be reliably reported by an 
adult household member responding to a 
Screener. 



Mailings to Respondents 

The literature on advance mailings to respondents 
in RDD surveys is quite sparse. It is likely that 
little has been done in this area because of the 
limited proportion of an RDD sample for which 
addresses can be obtained and the costs 
associated with the mailings. Cambum et al. 
(1995) report the results of an advance mailing 
for a national RDD survey about children's 
immunization. They found that mailing an 
advance letter in an RDD survey can lower the 
rate of refusal and improve overall response rates. 
No other pertinent work on advance letters for 
RDD surveys was located in the literature. 

Design of the Experiment 

Based upon information available in the extant 
literature and on experiences with previous 
NHES data collections, particularly the 
NHES:95, an experiment was developed and 
implemented as a part of a field test for the 
NHES:96. This research provided the 
opportunity to test the effects of the experimental 
conditions while holding constant the survey 
content, timing, interviewer training, and calling 
protocol. The experiment tested the effects of 1) 
screening out ineligible households before 
enumerating all household members and 2) 
sending respondents a letter about the survey 
prior to interviewer telephone contact. The field 
test sample was partitioned into four equal 
samples to examine the effects of four conditions: 

■ No advance letter, no screen-out question; 

■ No advance letter, screen-out question; 

■ Advance letter, no screen-out question; 
and 

■ Advance letter, screen-out question. 

The screen-out question used in this experiment 
asked the Screener respondent whether any of the 
people who normally lived in the household were 
age 20 or younger. If no one in the household 
was age 20 or younger, the Screener ended at this 
point. In households with members in the target 
age range, household enumeration was 
conducted. A copy of the questions in the 
Screener is given in appendix A. The box at the 
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bottom of the first page of the instrument shows 
that the screen-out interview ended if no 
members were in the eligible age range. If there 
were eligible household members or the 
household was part of the no screen-out sample, 
the interview continued in exactly the same way 
for both samples. The last part of the Screener 
contains the public library use questions. 

The advance letter used in the experiment 
identified the U.S. Department of Education as 
the sponsor of the survey, provided an overview 
of the purpose and content of the survey, stated 
the average amount of time required to complete 
interviews, and stressed the importance of 
participation while emphasizing that participation 
was voluntary. A page of commonly asked 
questions about the survey was also included. A 
copy of the advance letter and the other 
attachment are given in appendix B. These were 
mailed first class in a standard envelope with 
permit postage paid by the U.S. Department of 
Education. The letters were mailed 
approximately one week prior to beginning data 
collection. 

One hypothesis addressed in the experiment was 
that the lower than expected response rate 
observed in the NHES:95 was related to the full 
enumeration of all households even if no person 
in the household was selected for an extended 
interview. In the NHES:93, only households with 
children age 18 or younger or in 12th grade or 
below were fully enumerated. In the NHES:91, 
only households with children or those sampled 
for Adult Education (AE) component screening 
were enumerated, and only persons in the eligible 
age ranges (2 to 9; 16 or older) were enumerated. 
In each of these earlier studies, the interviewers 
could usually complete the interview quickly for 
households with no children, and this might have 
been used to encourage respondents to continue 
the interview, since it only involved answering a 
few items. This speculation led to the 
introduction of the screen-out part of the 
experiment for the field test. 

A second hypothesis was that the advance 
mailing would improve response rates. This was 
suggested primarily by experience with a 



nonrespondent mailing in the NHES:95, 
discussed above. Since the NHES:95 experience 
was not an experiment (obtaining an address and 
response propensity were confounded), the 
results were not conclusive. However, the 
NHES:95 experience strongly suggested that 
mailing a letter to respondents might be effective. 

In addition to these major hypotheses, several 
other results of the experiment were investigated 
to determine the consequences of surveying 
under the different conditions. One outcome 
measure studied was the percentage of 
households with children eligible for the Parent 
PFI/CI interview. A household was eligible if it 
had any member from age 3 through 12th grade 
and up to 20 years old (that is, preschoolers age 3 
and older and children enrolled in grades K 
through 1 2, up to a maximum age of 20). The 
percentages of households eligible for the Parent 
PFI/CI interview under the different conditions 
can be compared to estimates from other sources 
and serve as a rough indicator of the quality of 
the screening. 

Another aspect of the experiment examined is the 
resources required to conduct the interviews 
under the different conditions. The no screen-out 
interview is clearly longer for households with no 
members eligible for an extended interview 
because it contains more questions than the 
screen-out interview. However, these additional 
questions were added to increase the utility of the 
Screener. In the screen-out option, the Screener 
only serves to determine eligibility for the 
interview and to sample members for the 
extended interview components. In the no 
screen-out interview, the Screener is also used to 
obtain data on households for making national 
estimates. For example, in the NHES:96 the 
Screener was used to make national estimates of 
household public library use. 

A list-assisted sample of 10,001 telephone 
numbers was selected from the Genesys sampling 
frame used in the NHES in 1995 and 1996. The 
sample was systematically divided into two half- 
samples; one of the two halves was made 
available for interviewers at Westat's Rockville, 
Maryland Telephone Research Center (TRC) and 
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the other was made available to interviewers at 
Westat's Frederick, Maryland TRC. 

Screen-out condition. The screen-out question 
was included in the interviews conducted at the 
Frederick TRC, while all households were 
enumerated at the Rockville TRC. The screen- 
out question was restricted to one facility rather 
than assigning interviewers at both facilities to 
test this experimental condition in order to 
eliminate the contamination that might have 
occurred if interviewers at the same facility 
talked to one another about the study. The 
interviewers at the facilities were not informed 
that a different version of the interview was being 
tested at the other site. 

Although .this design eliminated the 

contamination between interviewers, it had the 
negative consequence of confounding the facility 
with the screen-out condition. Before this design 
was considered, the NHES:95 initial cooperation 
rates (defined in the next section) in the two 
facilities were compared and found to be nearly 
identical (60 and 61 percent). Westat strives to 
ensure there is no "facility effect" in their 
telephone centers by using standard training and 
supervision methods and monitoring the 

outcomes by center. 

Advance letter. For the advance letter 
component of the experiment, all 10,001 
telephone numbers were sent to Telematch, a 
commercial firm that matches telephone numbers 
to addresses. Rather than mail to all the 
addresses found, a random sample of half of the 
numbers assigned to each facility was selected 
and the advance letter was mailed only to the 
addresses obtained for those in the selected half- 
samples. Letters were sent via first class mail. 

Table 1 summarizes the design of the experiment 
and shows the number of telephone numbers 
assigned to each experimental condition, along 
with the results of sending the numbers to 
Telematch. Across all experimental conditions, 
advance letters could have been mailed to 33 
percent of the telephone numbers (3,348 of 



10,001), but under some experimental conditions, 
no letters were mailed. Of the 1,684 letters 
mailed, 184 (10.9 percent) were returned by the 
post office as undeliverable. Note that having an 
address associated with a telephone number does 
not necessarily mean that the correct household 
received the letter. Postmaster returns are 
accounted for in the analysis presented in this 
report, but it is likely that some additional letters 
were neither received by the households with the 
sampled telephone numbers nor returned. 

Data Collection Procedures 

The last step in the design was to select 
interviewers to conduct the interviews. 
Interviewers with experience working with the 
NHES:95 were selected in both facilities. 
Westat's telephone center management then 
examined the cooperation and refusal conversion 
rates of each of the interviewers from the 
NHES:95 to ensure that the interviewers in both 
facilities were comparable and that the selected 
interviews did not have unusually high rates. 
Twenty to 25 interviewers were trained at each 
telephone center. Following the training, the 
interviewing began. 

The sample of telephone numbers was sorted into 
random order and then released for work 
according to a standard call-scheduling protocol. 
This protocol required that at least seven attempts 
be made to contact a sampled telephone number 
and complete a Screener, with additional calls 
allocated to any extended interviews in the 
household. The calls were staggered at different 
times of the day and week, with two daytime 
weekday calls, two weekday evening calls, and 
two weekend calls on separate weekends. Initial 
refusal cases were held for a period of 13 days, at 
which time specially trained refusal conversion 
interviewers attempts to convert these cases. 
Unlike full-scale NHES surveys, no interviewing 
was done in Spanish for this experiment. The 
data collection procedures were identical in the 
two facilities. For example, if an interviewer 
reached an answering machine when a number 
was dialed, the call outcome was recorded and 



Table 1 . — Number of telephone numbers, by screen-out question and advance letter 



Experimental condition 


Number 


Percent 


Total 


10,001 


100.0 


Screen-out question 






Total 


5,001 


50.0 


Sampled for advance mail 


2,501 


25.0 


No mailable address 


1,661 


16.6 


Mailable address 


840 


8.4 


Mailed, returned 


90 


0.9 


Mailed, not returned 


750 


7.5 


Not sampled for advance mail 


2,500 


25.0 


No mailable address 


1,640 


16.4 


Mailable address 


860 


8.6 


No screen-out question 






Total 


5,000 


50.0 


Sampled for advance mail 


2,500 


25.0 


No mailable address 


1,656 


16.6 


Mailable address 


844 


8.4 


Mailed, returned 


94 


0.9 


Mailed, not returned 


750 


7.5 


Not sampled for advance mail 


2,500 


25.0 


No mailable address 


1,696 


17.0 


Mailable address 


804 


8.0 



NOTE: This table reflects the design of the field test experiment. As discussed in the text, 350 telephone numbers were removed 
from each half sample after the experiment began. The final sample numbers appear in table 2. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey (NHES), 
field test of the 1996 survey. 



no messages were left on the machine by the 
interviewers in either facility. The data collection 
period for the experiment lasted 5 weeks 

As the data collection continued, the number of 
completed interviews was monitored. After it 
was determined that most of the field test goals 
for the number of completed extended interviews 
of various types were being accomplished, it was 
decided to eliminate 350 of the telephone 
numbers in each facility that had not yet been 
worked at all. Since the sample was sorted 



randomly, this reduction in the sample was done 
very quickly without risk of bias. 

Table 2 gives the results of calling the telephone 
numbers by interviewing facility. This table 
provides some detailed information on the results 
of the call attempts. The counts for numbers that 
were not residential show the telephone numbers 
in the two facilities were very similar. The 
percentage of telephone numbers that were not 
residential in the two facilities were 



approximately equal (5 1 percent in the Frederick 
faculty and 52 percent in Rockville). 

Furthermore, the distributions of not residential 
numbers by type (not working, business, and not 
ascertained) were also very similar. The 
completion and . nonresponse outcomes are 
discussed in more detail in the Findings section. 

Outcome Measures 

Before discussing the results from the 
experiment, some terms are introduced that are 
needed in the evaluation of the experiment. The 
three types of measures used to evaluate the field 
test experiment are 1) cooperation, conversion, 
and response rates; 2) eligibility rates; and 3) 
resources expended to complete the interviews. 
Generally speaking, higher cooperation rates are 
the single most important evaluation criterion. 
Eligibility rates are an important consideration 
because they are one indicator of the extent of 
possible nonresponse bias in a survey. Resource 
measures reflect the cost associated with 
obtaining completed interviews; it is incumbent 
upon study designers to evaluate the effort 
required to achieve desired response rates and the 
resources required to support those efforts. 

Three cooperation rates and two response rates 
are examined in this experiment. The most 
important cooperation rate in the NHES is the 
Screener initial cooperation rate . When a 
household is contacted initially, the result can be 
either a completed screening interview, a refusal, 
or another result (such as a language problem or a 
temporary outcome like an appointment to call 
later). The initial contact with the household may 
be preceded by several call attempts that do not 
result in speaking to a person, such as a ring with 
no answer or an answering machine result. The 
initial cooperation rate is the ratio of the number 
of initially completed interviews (C) to the 
number of interviews that are initially either 
refused (R) or completed (C); thus, the rate is 
calculated as [C/(R+C)]. In some cases, the 
respondent neither refuses nor completes the 
interview, and a temporary outcome, such as an 
appointment to call later, is generated. These 
results are only included in the cooperation rate 



once they become either completes (C) or initial 
refusals (R). 

The Screener refusal conversion rate is a second 
cooperation rate used in this study. If a Screener 
interview is initially refused, an attempt to 
convert the refusal into a completed interview is 
tried some time later. The refusal conversion rate 
is the ratio of the number of these cases that are 
completed during refusal conversion (C 2 ) to the 
total number of interviews that were either 
refused a second time (R 2 ) or completed (C 2 ); 
thus the ratio is calculated as [C2/(R2 + C 2 )]. As 
noted below, only one refusal conversion was 
attempted in this experiment. However, this 
attempt might have required several telephone 
calls to contact the household. Other outcomes 
such as “maximum calls,” business, language 
problems, and nonworking numbers are not used 
in the computation. Hostile refusals, for which 
no conversion efforts were made, were also 
excluded from calculation of the conversion rate. 

The third type of cooperation rate discussed in 
this analysis is the extended interview 
cooperation rate ; the extended interviews are 
those conducted with sampled persons or parent 
respondents with sampled children within the 
screened households. Thus, it includes 
interviews sampled within households with 
completed Screeners. This rate is the ratio of the 
number of completed extended interviews (Cg) 
to the number of interviews that are refused (Rg) 
or completed (Cg); thus, the rate is calculated as 
[Cg/(Rg + Cg)]. If a sampled child was not at 
least 3 years old or under 21 and enrolled in 12th 
grade or less, the child was ineligible for the 
interview; these children are excluded from the 
computation of this rate. Outcomes such as 
“maximum calls,” language problems, and 
nonworking numbers are also excluded from this 
rate. No attempts were made to convert the 
extended interviews that were initially refused 
during this experiment. 1 



1 NHES field tests typically do not include refusal 
conversion attempts. Such attempts were made at the 
Screener level for this experiment, but not at the extended 
interview level. 
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Table 2. — Number of telephone numbers dialed, by final Screener response category and screen-out 
question 



Screener response category 


Screen-out question 
(Frederick) 


No screen-out question 
(Rockville) 




Number 


Percent 


Number 


Percent 


Total 


4,651 


100.0 


4,650 


100.0 


Complete 


1,721 


37.0 


1,381 


29.7 


Complete, no extended interview 


1,213 


26.1 


877 


18.9 


Complete, Parent PFI/CI interview only 


226 


4.9 


238 


5.1 


Complete, Parent PFI/CI and Youth Cl 
interviews 


282 


6.1 


266 


5.7 


Not residential 


2,357 


50.7 


2,417 


52.0 


Not working 


1,405 


30.2 


1,459 


31.4 


Business 


553 


11.9 


540 


11.6 


Not ascertained 


399 


8.6 


418 


9.0 


Nonresponses, finalized 


430 


9.2 


559 


12.0 


Language problem 


29 


0.6 


37 


0.8 


Maximum call 


115 


2.5 


130 


2.8 


Refusal 


284 


6.1 


377 


8.1 


Other nonresponse 


2 


0.0 


15 


0.3 


Nonresponse, not finalized 


143 


3.1 


293 


6.3 


Language problem 


10 


0.2 


25 


0.5 


Maximum call . 


47 


1.0 


197 


4.2 


Refusal 


86 


1.8 


71 


1.5 



SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey (NHES), 
field test of the 1996 survey. 



Exhibit 1 . — Definitions of Screener results for calculation of outcome measures 



Result 


Definition 


C 


Completed Screener with no initial refusal 


R 


Initial Screener refusal 


c 2 


Screener completed in refusal conversion 


r 2 


Screener refused again in refusal conversion 


C e 


Completed extended interview 


Re 


Refused extended interview 




1 K 
iL 0 
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An overall or gross Screener response rate is also 
used to evaluate some of the results from the 
experiment. The gross response rate is the ratio of 
the number of completed interviews to the 
number of known residential numbers. While the 
gross response rate is used in most NHES studies, 
the field test gross response rate is not very 
indicative of the expected Screener response rate 
for a full-scale NHES for several reasons. In a 
full-scale NHES, data are collected over a longer 
period of time and this generally increases the 
percentage of households that are contacted and 
the response rate. A full scale study would also 
include interviews conducted in Spanish to help 
reduce the percentage of language problem cases. 
Furthermore, the call scheduling protocol used in 
the field test would be replaced by more 
extensive dialings to telephone numbers that 
could not be interviewed and refusal conversion 
would be attempted for some cases a second 
time. The use of these more intensive methods in 
a full scale study, a standard for the NHES, 
reduces the number of unresolved numbers and 
increases the gross Screener response rate. 
Despite the shortcomings of this measure, the 
gross Screener response rate is used in a limited 
way to compare the results across experimental 
conditions. 

Because the gross Screener response rate for the 
experiment is not very useful for predicting the 
actual response rate in a full scale study, an 
alternative, called the predicted Screener 
response rate , is introduced. The predicted 
Screener response rate is the initial cooperation 
rate for the Screener multiplied by one minus the 
Screener refusal conversion rate. Let / be the 
initial cooperation rate for the Screener and let r 
be the refusal conversion rate for the Screener, 
then the predicted response rate is 100{/+(l-/)*r} 
percent. 

For example, in the NHES:93 the initial 
cooperation rate was 69.7 percent and the refusal 
conversion rate was 45 percent, so the predicted 
Screener response rate was 83.3 percent 

( 1 00* { .697+( 1 -.697)* .45 } =83 .3 %). The actual 
Screener response rate in the NHES:93 was 82.1 
percent. For the NHES:95, the predicted 
Screener response rate was 74.2 percent 



0 




(100*{.61+(1-.61)*.339}= 74.2%) and the actual 
Screener response rate was 73.3 percent. In both 
cases the predicted Screener response rate is 
about 1 percent higher than the actual rate. Thus, 
the predicted response rate provides a reasonable 
predicted value using only initial cooperation and 
refusal conversion rates. This rate is particularly 
useful because the initial cooperation and refusal 
conversion rates do not depend on the calling 
protocol (number of call attempts, maximum 
number of attempts, number of times refusal 
conversions are attempted, etc.), and the calling 
protocol for the experiment differs from the 
standard NHES protocol. The predicted Screener 
response rate is robust to these differences in 
protocols. 

The eligibility rate can be another useful statistic 
in assessing the survey outcome. The eligibility 
rate is the percentage of the respondents who are 
the target of the survey. For example, if the goal 
is to conduct interviews with parents about 
children and youth age 3 through 12th grade, then 
the eligibility rate is the percent of households 
with children in this population. Examining the 
eligibility rate is a means of assessing the extent 
to which the cooperating respondents reflect the 
population on the key characteristic that 
determines eligibility for the study. Significant 
deviation of this rate from known population 
parameters suggests nonresponse bias at the 
screening level. 

The measures of resources that are used to 
evaluate the experiment are the number of 
telephone calls, the length of the Screener 
interview, and the number of air time interviewer 
hours. The length of the interview is the 
administration time for the interview. It begins 
when the respondent answers the telephone and 
ends when the screener is completed. The air 
time is the amount of time used to make the calls, 
get the respondent on the telephone, conduct the 
interview, and code the result of the call; it 
includes both those attempts resulting in contact 
and those not resulting in contact. These 
measures are provided at the Telephone Research 
Center level so that they can be used to compare 
the screen-out question condition of the 
experiment but not the advance letter factor. 
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Table 3. — Number of Screener, Screener refusal conversion, and extended interview cases, by cooperation 
rates, screen-out question, and advance letter 



Experimental condition 


Screener* 


Screener refusal conversion 


Extended interview 


Complete 


Refusal 


Cooperation 

rate 


Complete 


Refusal 


Conversion 

rate 


Complete 


Ineligible 


Refusal 


Cooperation 

rate 


Screen-out/letter 


802 


272 


74.7% 


67 


144 


31.8% 


342 


6 


17 


95.3% 


Screen-out/no letter 


769 


269 


74.1 


83 


135 


38.1 


284 


10 


23 


92.7 


No screen-out/letter 


671 


348 


65.8 


71 


179 


28.4 


299 


9 


21 


93.6 


No screen-out/no letter 


572 


387 


59.6 


67 


196 


25.5 


255 


11 


15 


94.7 


Screen-out 


1,571 


541 


74.4 


150 


279 


35.0 


626 


16 


40 


94.1 


No screen-out 


1,243 


735 


62.8 


138 


375 


26.9 


554 


20 


36 


94.1 


Letter 


1,473 


620 


70.4 


138 


323 


29.9 


641 


15 


38 


94.5 


No letter 


1,341 


656 


67.2 


150 


331 


31.2 


539 


21 


38 


93.6 



*At this stage, refusal conversion attempts are not included. 



SOURCE: U S. Department of Education, National Center for Education Statistics, National Household Education Survey (NHES), 
field test of the 1996 survey. 



The costs of sending an advance letter are 
dependent on a number of factors that may vary 
from one survey to the next. In this study, 
Telematch charged $0.05 for each address found 
for a telephone number rather than charging per 
telephone number searched. Other types of 
arrangements with commercial firms who do 
these services might be arranged, especially for 
large volume mailings. 

Other components of resources are the costs for 
printing the advance letter, printing the 
envelopes, preparing the mailing, and postage. In 
this study, a simple letter was prepared (see 
appendix B) and the postage was paid by the U.S. 
Department of Education. As with the methods 
for obtaining addresses, different arrangements 
for conducting the mailing, such as using a 
company that specializes in these activities, or 
using regular stamps rather than government 
permit, and a different number of mailings could 
result in overall costs varying substantially. 
Because of these variations, the focus in the next 
section is on resources used in the interviewing 
rather than the mailing. These are also clearly the 
predominant expenditures in this type of survey. 



Findings 

The findings of the experiment as they relate to 
the screening cooperation, conversion, and 
response rates, eligibility rates, and survey 
resources requirements are presented below. 

Cooperation, Conversion, and Response 
Rates 

Table 3 shows the number of Screener cases prior 
to refusal conversion, Screener refusal conversion 
cases, and extended interview cases for each of 
the four experimental conditions, as well as 
aggregates for each of the two conditions. The 
cooperation rates are computed using the methods 
described above. 

As noted above, the initial cooperation rates are 
the most important in any RDD survey, and the 
differences in these rates between the 
experimental conditions are obvious. The initial 
Screener cooperation rates for the screen-out 
conditions are much higher than for the no 
screen-out condition. 
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Table 4 presents the cooperation rates and their 
standard errors. It also gives the t- value for each 
combination of experimental conditions, where 
the t- value is the ratio of the rate to its standard 
error. The t-values in bold are statistically 
significant at the 95% confidence level after 
controlling for the multiple tests being performed 
(Bonferroni adjustment with family size 7). 

For the Screener initial cooperation rates, the 
results are striking. The difference in rates 
between the screen-out and no screen-out 
conditions is 11.5 percent (e-f). The screen-out 



condition is the dominant factor for determining 
the level of the initial cooperation rate for the 
Screener in this experiment. The difference in 
the initial Screener cooperation rate due to the 
advance letter is 3.2 percent (g-h), but this 
difference is significant only for those interviews 
conducted with the no screen-out question (c-d). 
The screen-out question leads to a substantially 
higher initial cooperation rate. If no screen-out 
question is included, then the advance letter 
increases the initial cooperation rate (c-d), but not 
to the same level attained by using the screen-out 
question (a-c). 



Table 4. — Differences in cooperation rates, by screen-out question and advance letter 



Experimental condition 


Screener cooperation* 


Screener refusal conversion 


Extended interview 


Rate 


s.e. 


t-value 


Rate 


s.e. 


t-value 


Rate 


s.e. 


t-value 


(a) Screen-out/letter 


74.7% 


1.3 




31.8% 


3.2 




95.3% 


1.1 




(b) Screen-out/no letter 


74.1 


1.4 




38.1 


3.3 




92.7 


1.5 




(c) No screen-out/letter 


65.8 


1.5 




28.4 


2.9 




93.6 


1.3 




(d) No screen-out/no letter 


59.6 


1.6 




25.5 


2.7 




94.7 


1.3 




Difference (a)-(b) 


0.6 


1.9 


0.3 


-6.3 


4.6 


-1.4 


2.6 


1.8 


1.4 


Difference (a)-(c) 


8.8 


2.0 


4.4 


3.4 


4.3 


0.8 


1.7 


1.7 


1.0 


Difference (a)-(d) 


15.0 


2.1 


7.3 


6.3 


4.2 


1.5 


0.7 


1.7 


0.4 


Difference (b)-(d) 


14.4 


2.1 


6.9 


12.6 


4.2 


3.0 


-1.9 


2.0 


-1.0 


Difference (c)-(d) 


6.2 


2.2 


2.9 


2.9 


3.9 


0.7 


-1.0 


1.9 


-0.5 


(e) Screen-out 


74.4 


0.9 




35.0 


2.3 




94.1 


0.9 




(f) No screen-out 


62.8 


1.1 




26.9 


2.0 




94.1 


1.0 




Difference (e)-(f) 


11.5 


1.4 


8.0 


8.1 


3.0 


2.7 


0.0 


1.3 


0.0 


(g) Letter 


70.4 


1.0 




29.9 


2.1 




94.5 


0.9 




(h) No letter 


67.2 


1.1 




31.2 


2.1 




93.6 


1.0 




Difference (g)-(h) 


3.2 


1.4 


2.2 


-1.3 


3.0 


-0.4 


0.9 


1.3 


0.7 



*At this stage, refusal conversion attempts are not included. 



NOTE: t-values equal to or greater that 2.7 are significant at the 95 percent confidence level. Calculations of differences are based 
on unrounded data. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey 
(NHES), field test of the 1 996 survey. 
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In addition to the differences in the cooperation 
rates, a substantial difference was observed 
between the screen-out and no screen-out 
conditions in the numbers of nonfinalized 
maximum call cases (see table 2). While only 47 
cases were in this status in the screen-out 
condition, there were 197 such cases in the no 
screen-out condition. This may be a function of 
the relative ease of completing the Screener with 
ineligible households in the screen-out condition 
compared to the no screen-out condition. Another 
factor may be that some maximum call cases are 
hidden refusals. In the NHES:95, 41 percent of 
maximum call cases that were refielded for 
additional attempts were finalized as refusals 
(Collins et al. 1997). The greater incidence of 
maximum call cases in the no screen-out 
condition, is consistent with higher rate of refusal 
for that condition. 

For the Screener refusal conversion rates, only 
two of the differences are statistically significant. 
The screen-out question not only results in a 
higher Screener initial cooperation rate, but it also 
has the desirable feature of resulting in a higher 
refusal conversion rate. The increase of 8.1 
percent is large enough to be of substantive 
importance. It is worth noting that most of the 



difference in refusal conversion rates in the 
screen-out question occurs when no advance 
letter is mailed. 

No statistically significant differences were 
observed for the extended interview initial 
cooperation rates. All of these rates were 
uniformly high. 

A goal of the experiment was to examine the 
probable Screener response rate under these 
different experimental conditions. The predicted 
Screener response rate shows this better than any 
other single measure, although the rates are about 
1 percent higher than the actual response rates. 
This rate represents the expected Screener 
response rate under the full NHES data collection 
protocol; it is discussed further on page 10. Table 
5 shows the predicted Screener response rates for 
each of the experimental conditions. The 
predicted Screener response rates using the 
screen-out question are about 82 to 84 percent, 
while enumerating all households gives predicted 
Screener response rates in the range of 70 to 75 
percent. The gross Screener response rate can be 
used to measure the difference in response rates 
by whether or not an advance letter was actually 



Table 5. — Predicted Screener response rate, by screen-out question and advance letter 



Experimental condition 


Number of 
telephone 
numbers 


Initial 

cooperation 

rate 


Refusal 

conversion rate 


Predicted 
response rate 


Screen-out/letter 


2,326 


74.7% 


31.8% 


82.7% 


Screen-out/no letter 


2,325 


74.1 


38.1 


84.0 


No screen- out/letter 


2,325 


65.8 


28.4 


75.5 


No screen-out/no letter 


2,325 


59.6 


25.5 


69.9 


Screen-out 


4,651 


74.4 


35.0 


83.3 


No screen-out 


4,650 


62.8 


26.9 


72.8 


Letter 


4,651 


70.4 


29.9 


79.2 


No letter 


4,650 


67.2 


31.2 


77.4 



SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey (NHES), 
field test of the 1996 survey 
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mailed to a household. The gross Screener 
response rate, as noted earlier, is the ratio of the 
number of completed interviews to the number of 
residential numbers. This rate is the only one 
used for examining the detailed results because 
the other rates, especially the predicted rate, 
become more unstable when based on fewer 
cases. For the purpose of examining the impact 
that mailing the letter has on the Screener 
response rate, the gross Screener response rate is 
used. As noted earlier, the fact that a letter was 
mailed does not mean that it was delivered to any 
household, that it was delivered to the correct 
household, or that the Screener respondent ever 
saw the letter. Interviewers were aware that the 
mailing to some households was conducted, but 
had no information about whether specific 
households had been sent a letter. The 
information available is whether or not the 
advance letter was mailed and, if so, whether it 
was returned by the postmaster as undeliverable. 
The postmaster returns are only available for 
those advance letters that were in the part of the 
experiment in which the letters were actually 
mailed. 

Table 6 presents the number of households and 
the gross Screener response rates by the various 
experimental conditions. The gross Screener 
response rate shows the large difference in 
response rates associated with the use of the 
screen-out question, as did the cooperation and 
conversion rate measures discussed above. Under 
the screen-out condition, no significant difference 
in Screener response rates is observed between 
households who were and were not sampled for 
mailing (74.8 and 75.3 respectively). The 
comparison of these rates examines whether there 
is an improvement in response rates when an 
advance mailing is part of the data collection 
protocol, and not whether households with 
mailable addresses respond at different rates 
regardless of whether a letter is sent. The latter 
question is of less interest, but is discussed below. 
Under the no screen-out condition, those who 
were sampled for mailing had a 5.9 percent 
higher Screener response rate (64.7 percent) than 
those who were not sampled for mailing (58.8 
percent). These findings suggest that the advance 
mailing may help to increase Screener response 



rates under a no screen-out design, but not under 
a screen-out design. 

Although the main issue is the effect of doing an 
advance mailing, it is interesting to examine the 
difference in Screener response rates for those 
with mailable addresses under the two conditions. 
Under the screen-out conditions, the difference 
between the gross Screener response rates for 
those sampled for the advance letter with a 
mailable address (79.5 percent) and those not 
sampled for the advance letter with a mailable 
address (78.0 percent) was only 1.5 percent (t = 
0.7). Under the no screen-out condition the 
difference between the two groups was 
significant at 7.4 percent (t = 2.7). Thus, there 
appears to be an interaction between the two 
experimental conditions and the effect of the 
mailing is different depending on the use of the 
screen-out item. 

The lower Screener response rates associated 
with the no screen-out approach raise the 
important question of the source of the loss in 
response. As noted previously, the literature 
generally regards enumeration as more invasive 
than other respondent selection approaches. 
However, data collection experience on the 
NHES indicates that most refusals occur prior to 
the enumeration matrix (Brick et al., 

forthcoming). Some researchers (Bercini and 
Massey 1979) have suggested that the lower 
response rates may be related to the 

interviewers’ perception of the task. 

Specifically, Bercini and Massey reported that 
collecting household members’ names had a 
“direct negative effect on interviewer 

performance.” The findings of this experiment 
are consistent with that hypotheses and are 
further supported by the fact that most refusals 
occur prior to enumeration, and cannot be 
attributed to respondent reaction to the 
enumeration (Brick at al. 1997). Following this 
screening experiment, efforts were made in the 
NHES:96 interviewer training to emphasize that 
only a brief interview (screening questions, 
personal characteristics, and library use 
questions) was to be conducted in most 
households. However, the NHES:96 Screener 
response rate was at the lower end of the range 



Table 6. — Number of households and gross Screener response rate, by screen-out question and advance 
letter 



Experimental condition 


Number of 
households 


Gross response 
rate 


t-value 


Total 


4,527 


68.5 




No mailable address 


1,965 


64.5 




Mailable address 


2,562 


71.6 




Screen-out question 








Total 


2,294 


75.0 




(a) Sampled for advance mail 


1,162 


74.8 




No mailable address 


527 


69.1 




(b) Mailable address 


635 


79.5 




Mailed, returned 


52 


76.9 




Mailed, not returned 


583 


79.8 




(c) Not sampled for advance mail 


1,132 


75.3 




No mailable address 


463 


71.3 




(d) Mailable address 


669 


78.0 




Difference (a) - (c) 




-0.5 


-0.3 


Difference (b) - (d) 




1.5 


0.7 


No screen-out question 








Total 


2,233 


61.8 




(e) Sampled for advance mail 


1,147 


64.7 




No mailable address 


499 


60.7 




(0 Mailable address 


648 


67.7 




Mailed, returned 


52 


65.4 




Mailed, not returned 


596 


68.0 




(g) Not sampled for advance mail 


1,086 


58.8 




No mailable address 


476 


56.9 




(h) Mailable address 


610 


60.3 




Difference (e) - (g) 




5.9 


2.9 


Difference (f) - (h) 




7.4 


2.7 



NOTE: t-values equal to or greater that 2.7 are significant at the 95 percent confidence level. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey 
(NHES), field test of the 1996 survey. 
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expected for a no screen-out approach with an 
advance mailing (the NHES:96 Screener 
response rate was 70 percent). Thus it appears 
that direct experience with conducting a given 
survey shapes the interviewers’ perception of 
the ease or difficulty of the task, which in turn 
has an effect on response rates. 

Eligibility Rates 

As stated earlier, differences in eligibility rates 
could indicate potential response problems. 
Based on the October 1992 CPS (the most recent 
October CPS file available at the time of the 
initial NHES:96 sample design), the percentage 
of households eligible for the NHES:96 was 
estimated to be 30.4 percent. The percentage of 
households eligible for the NHES:96 based on the 
completed Screener interviews using the screen- 
out question was 29.5 percent (508 households 
out of 1,721 completed Screeners), within 1 
percent of the CPS estimate. The percentage in 
the numbers assigned to the interviews with no 
screen-out question was 36.5 percent (504 
households out of 1,381 completed Screeners), 
about 6 percent higher than the CPS estimate. 
(t=2.52). This result initially led to speculation 
about potential nonresponse biases due to 
differential response rates by whether or not there 
were any children in the household. However, 
further examination of the estimates from the 
CPS reduced the concern, because the CPS 
estimates were found to vary over time. The 
estimated percentage of households with eligible 
children from the October 1992 CPS was 30 
percent, while the October 1993 CPS estimate 
was 33 percent and the October 1994 CPS 
estimate was 38 percent. Since the eligibility 
rates under the two experimental conditions were 
within the range of the CPS for the three years, it 
is difficult to determine which of the 
experimental rates is closer to the population 
value. 

The difference in the eligibility rates by 
experimental condition may be less problematic 
than they first appeared for another reason. Brick 
and Broene (forthcoming) found that the calling 
procedures used in the full scale NHES surveys 
to increase response rates (e.g., second refusal 



conversion attempts, additional attempts to 
contact households that did not respond in the 
first 7 callings) virtually eliminated the eligibility 
differences that could be measured. Specifically, 
comparisons of the characteristics of households 
from the NHES:95 and from the Current 
Population Survey revealed relatively few 
differences in characteristics associated with 
eligibility, and the observed differences were 
generally quite small. Exceptions were 1 -person 
households (for which the NHES:95 estimate was 
2.1 percent less than the CPS estimate) and 
households with at least one adult aged 65 or 
older (2.7 percent less than the CPS). Thus, the 
statistically significant eligibility rates found in 
the experiment would probably be reduced if the 
standard NHES calling procedures were used. 

Resources 

The other part of this evaluation is the 
consideration of. three measures of resources 
expended to conduct the surveys: 1 ) the number 
of telephone calls made, 2) the length of the 
interview, and 3) the air time required. As noted 
earlier, these measures are provided at the facility 
level so that differences between the screen-out 
question and the full enumeration can be 
evaluated. The difference due to the advance 
letter cannot be examined, but this is only a 
minor restriction since the advance letter was not 
expected to reduce the length of the interview. 
The advance letter could have an impact on 
reducing the number of calls needed to finalize a 
case and this can be evaluated. 

The mean number of telephone calls per case 
made in the field test did not differ substantially 
by the use of the screen-out question. The mean 
number of calls with the screen-out sample was 
2.76 and the average number of calls with no 
screen-out question was 2.83. The difference is 
small and of little practical consequence for a 
survey like the NHES. About 6 percent of the no 
screen-out interviews were not finalized and only 
3 percent of the screen-out interviews were not 
finalized. It is likely that the mean number of 
telephone calls required to finalize the cases so 
that the percentages finalized were equal would 
have been somewhat greater for the no screen-out 



numbers. However, the differences are probably 
small, because similar results are obtained if the 
mean number is computed separately for only the 
completed household screening interviews (2.69 
for the screen-out question and 2.73 for no 
screen-out question). 

The average length or administration time of the 
Screener differed by whether or not the screen- 
out question was used. The Screener contained 
questions on the educational and demographic 
characteristics of household members and items 
on public library use 2 . These questions were 
asked in all households in the no screen-out 
condition, but were only asked in the screen-out 
condition if there were household members age 
20 or younger and the Screener was continued. 
The mean times for households that were not 
sampled for an extended interview were very 
different for these two screening conditions. The 
mean for households not sampled for an extended 
interview with the screen-out question was 2.1 
minutes and for those without the screen-out 
question the mean was 8.1 minutes (table 7). The 
very brief time for those without extended 
interviews in the screen-out condition reflects the 
fact that most of these households were screened 
out with only a few questions. While these 
differences have cost implications, it must be 
remembered that much of the additional survey 
administration time is associated with collecting 
data on the characteristics of households and their 
members that would not be available for analysis 
otherwise. The difference in cooperation rates 
discussed earlier suggests that some of the 
additional time is required for refusal conversion 
activities, i.e., persuading persons who initially 
refused to complete the interview. However, 
instrument administration time alone is not an 
appropriate comparison of resources associated 
with the screening experiment, since more data 
were collected under the no screen-out condition. 



2 The questions on public library use were asked in the 
Screener if no household members were sampled for an 
extended interview; if one or more household members 
were sampled for extended interviews, the public library 
use questions were asked during the first extended 
interview in the household. 



The survey administration time discussed above 
reflects effort expended on completed cases. Air 
time reflects interviewing effort for both 
responses and nonresponses. The total air time 
was 520 hours in the Telephone Research Center 
with the screen-out question and 645 hours in the 
Center without the screen-out question. This 
difference is primarily due. to two factors: the 
length of the interview for households with no 
extended interviews (which was reflected in the 
survey administration time discussed above) and 
additional time for refusal conversion efforts. The 
increase is nearly 25 percent above the air time 
devoted to the telephone numbers that had the 
screen-out question. Since air time is probably 
more highly correlated to the total data collection 
cost of the survey than any other single measure, 
it is reasonable to speculate that the cost of doing 
the no screen-out method and collecting the 
household-level data (demographic, educational, 
and library items) for all households is about 25 
percent greater than the cost of conducting data 
collection with a screen-out question under these 
or similar eligibility conditions. 

Conclusions 

The results of this experiment were definitive. 
The use of a screen-out question to eliminate 
ineligible households prior to enumeration results 
in considerably higher response rates. The cost 
of interviewing is also about 25 percent lower for 
this approach if the items included in the 
expanded screening interview are similar to those 
used in the NHES:96 and about one-third of all 
households had members eligible for extended 
interviews. Some of the additional resources 
required for the no screen-out condition are 
associated with the collection of these additional 
data. However, the cost differential is also 
associated with the need for a larger refusal 
conversion effort. 

In the wake of the lower than planned response 
rates from the NHES:95, the screen-out question 
approach could be used to obtain high response 
rates in future NHES collections and other 
surveys in which only a subset of households is 
eligible for the extended component interviews. 
Since the advance letter does not increase 
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Table 7. — Administration time of Screeners, by experimental condition and household type 



Experimental condition and household type 


Number 




Time in minutes 








Mean 


Minimum 


Maximum 


Screen-out question 

No extended interviews 


1,213 


2.1 


0.7 


23.4 


Parent PFI/CI interview only 


226 


5.3 


2.9 


13.2 


Parent PFI/CI and Youth Cl interviews 


282 


5.9 


2.9 


16.0 


No screen-out question 

No extended interviews 


877 


8.1 


3.6 


35.7 


Parent PFI/CI interview only 


238 


5.6 


2.8 


24.7 


Parent PFI/CI and Youth Cl interviews 


266 


6.5 


3.3 


15.9 



SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey (NHES), 
field test of the 1996 survey. 



response rates when the screen-out approach is 
used, its use under a screen-out design should not 
significantly increases response rates. However, 
a letter might still be useful for refusal or other 
nonresponse conversion purposes under these 
conditions. 

Despite the advantage of a higher response rate, 
the screen-out approach is not acceptable if 
estimates of the characteristics for all households 
are necessary to meet the goals of the study. The 
experiment showed that mailing an advance letter 
was effective in increasing the Screener response 
rate when the full enumeration of all households 
was conducted. Thus, the advance mailing 
should be used in such designs. 

Under a full enumeration design, other procedural 
methods besides the advance mailing should be 
considered. As noted earlier, some researchers 
(Bercini and Massey 1979) have suggested that 
the lower response rates may be related to the 
interviewers' perceptions of the task. If this is 
true, then training methods that emphasize the 
simplicity of the job might improve response 
rates. For the NHES:96, the no screen-out method 
was used . and only personal and household 
characteristics and a few library items were 
collected in the 70 percent of households that did 
not have extended interviews. The interviewers 
in the NHES:96 were trained to expect a short 



interview for these households. Despite this 
training approach, the Screener response rate for 
the NHES:96 was 70 percent which is in the 
lower end of the range predicted based on the 
results of this experiment. It is likely that 
interviewer’s actual experience in conducting the 
survey overrides messages conveyed in training 
about the ease or difficulty of the task. Thus, 
training may not be adequate to overcome the 
lower response rates. 

Other procedural methods used in the NHES have 
been shown to improve response under both 
screen-out and full enumeration designs. These 
include second refusal conversion attempts and 
refielding nonresponse cases such as those with 
maximum numbers of contact attempts and those 
with only noncontact (e.g., answering machine) 
results. Extending the data collection period to 
allow refusals to be held for a longer period prior 
to attempting conversions and to provide a longer 
period over which to contact other nonresponse 
cases may be helpful, but this approach needs to 
be considered in terms of the attendant costs 
versus benefits and the time constraints under 
which the survey must operate. 

During the development phase of a survey, it is 
incumbent upon the researchers to consider the 
balance between the substantive goals of the 
study and the response and resource implications 
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of the designs that could be used to meet those 
substantive goals. This experiment indicates that 
the approach to enumeration used in an RDD 
survey has implications both for the screening 
response rate and for the resources required to 
carry out the survey. 
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Appendix A: 

NHES:96 Field Test Screener 



NHES:96 Field Test Screener 



SCRN 



Si . Hello, this is (interviewer) and I'm calling about a research study sponsored by the 

United States Department of Education. Are you a member of this household 
and at least 18 years old? 

HHMEMB18 

YES 1 

NO 2 

PROBABLE BUSINESS 3 

GO TO RESULT GT 



(GO TO S4) 
(GO TO S2) 
(GO TO S4) 



S2. May I please speak with a household member who is at least 1 8 years old? 



SPEAKMEM 



AVAILABLE 1 

NOT AVAILABLE 2 

THERE ARE NONE 3 

GO TO RESULT GT 



(GO TO SI) 

(GO TO RESULT, CALLBACK 
APPT.) 

(GO TO S3A) 



S3A. May I please speak with the male or female head of this household? 



SPEAKHH 



PERSON ON PHONE ' 

OTHER PERSON, AVAILABLE 2 

OTHER PERSON, NOT AVAILABLE 3 

GO TO RESULT GT 



(GO TO S4) 

(GO TO S3 B) 

(GO TO RESULT, CALLBACK 
APPT.) 



S3B. Hello, this is (interviewer) and I'm calling about a research study sponsored 

by the United States Department of Education. Are you a head of this household? 



HHMEMBHH 

...1 (GO TO S4) 
...2 (GO TO S3A) 
GT 

S4. Is this phone used for... 



YES 

NO 

GO TO RESULT. 



SFONEUSE 



Home use 1 

Home and business use, or 2 

Business use only? 3 

GO TO RESULT GT 



(CONTINUE) 

(CONTINUE) 

(GO TO THANKl ) 



For Parent/Youth Interview, read SCRN_20. 
For Adult Cl Interview, read PRE_UB. 




25 2S 



SCRN 20. 



STMT 1. 
STMT 2. 
PRE_LIB. 

PRE_S6. 

S6. 



S6VERF1. 



The U.S. Department of Education is conducting a voluntary and confidential study about the 
educational experiences of children and how they learn about their communities and government. 
We are also interested in how all households use public libraries. Are any of the people who 
normally live in your household age 20 or younger? 

YES 1 (readSTMTI) 

N0 2 (read STMT2) 

GO TO RESULT QT 



Split-half 1: If SCRN_20 = 2, then terminate, else continue. 
Split-half 2: Continue. 



These next questions are about the people in your household and usually take 5 to 7 minutes 
(GO to S6.) 



Then I just have a few questions about libraries and about the people in your household. They 
usually take 5 to 7 minutes, (go to LINTRO.) 

The U.S. Department of Education is conducting a voluntary and confidential study about how 
people use public libraries and how they learn about their communities and government. These 
questions usually take 1 0 to 1 5 minutes. 

First, I’d like to ask about libraries, (goto LINTRO.) 

Now I d like to ask about the people in your household and their education and background. 



ENUM 

Starting with yourself, please tell me just the first names and ages of all the people who normally live 
in your household. What is your first name, please? 

[HOUSEHOLD members include people who think of this household as their primary place of 

RESIDENCE. IT INCLUDES PERSONS WHO USUALLY STAY IN THE HOUSEHOLD BUT ARE TEMPORARILY AWAY 
ON BUSINESS, VACATION, IN A HOSPITAL, OR LIVING AT SCHOOL IN A DORM, FRATERNITY, OR SORORITY.] 



What is 
[your first 
name/the first 
name of the 
next person]? 


How old 
[are you/ 
is (he/she)]? 


Is this person 
male or 
female? 


SCREENER 

RESPONDENT 


FNAME 


AGE 


SEX 


screspx 



[VERIFY THE NUMBER OF HOUSEHOLD MEMBERS LISTED ON THE MATRIX.] Have We missed 
anyone else who usually lives here who is temporarily away from home or living 
in a dorm at school, or any babies or small children? 

MATRIX CORRECT -| 

RETURN TO MATRIX 2 

GOTO RESULT QT 



If AGE >= 3, -7, -8 (person age 3 or older), ask SX7. 
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SX7. 



[Are you/ls (person)] attending (or enrolled in) (school/nursery school, 
kindergarten, or school)?] 



SENROLL 

YES 1 

NO 2 

REFUSED -7 

DON'T KNOW *8 



If AGE >= 18 or -7, -8 (person age 18 or older), autocode SX8 = 
2 (not home schooled) and go to box after SX8. Else, if AGE = 
5-17, ask SX8. Else, if SX7 = 1 (person is enrolled in school), 
go to SX9. Else, go to first box after SX12. 



SX8. (read first time: Some parents decide to educate their children at home rather 

than sending them to school.) Is (child) being schooled at home? 



SHOMESCH 



YES 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 



(GO TO SX8A) 

(GO TO BOX AFTER SX8A) 
(GO TO BOX AFTER SX8A) 
(GO TO BOX AFTER SX8A) 



SX8A. So your child is being schooled at home instead of at school? 



SHOMECON 



YES 


i 


NO 


2 


REFUSED 


-7 


DON'T KNOW 


-8 



(GO TO SX10) 

(RECODE SX8=2 AND GO TO 
GO TO BOX) 

(RECODE SX8 = -7 AND 
GO TO BOX) 

(RECODE SX8 = -8 AND 
GO TO BOX) 



If SX7 = 1 (person is enrolled in school), go to SX9. 
Else, go to first box after SX12. 



ENUM 
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SX9. 



What grade or year of school [are you/is (person)] attending? 
[probe for t or p: Is that before or after kindergarten?] 



SGRADE 

NURSERY/PRESCHOOL/PREKINDERGARTEN/HEAD START N (GO TO SX1 1 ) 

TRANSITIONAL KINDERGARTEN (BEFORE K) T (GO TO SX1 1 ) 

KINDERGARTEN « (GOTOSX11) 

PREFIRST GRADE (AFTER K) p (GOTOSX11) 

FIRST GRADE 1 (GOTOSX11) 

SECOND GRADE 2 (GO TO SX1 1 ) 

THIRD GRADE 3 (GO TO SX1 1 ) 

FOURTH GRADE 4 (GO TO SX1 1 ) 

FIFTH GRADE 5 (QO TO SX1 1 ) 

SIXTH GRADE 6 (GO TO SX1 1 ) 

SEVENTH GRADE 7 (GO TO SX1 1 ) 

EIGHTH GRADE 8 (GO TO SX1 1 ) 

NINTH GRADE/FRESHMAN IN HIGH SCHOOL 9 (GO TO SX1 1 ) 

TENTH GRADE/SOPHOMORE IN HIGH SCHOOL 10 (GO TO SX1 1 ) 

ELEVENTH GRADE/JUNIOR IN HIGH SCHOOL 1 1 (GO TO SX1 1 ) 

TWELFTH GRADE/SENIOR IN HIGH SCHOOL 12 (GO TO SX1 1 ) 

UNGRADED ELEMENTARY/SECONDARY U (GO TO SX1 0) 

SPECIAL EDUCATION S (GO TO SX1 0) 

VOCATIONAL/TECHNICAL AFTER HIGH SCHOOL 1 5 (GO TO SX9A) 

COLLEGE (UNDERGRADUATE) 16 (GO TO SX9B) 

GRADUATE, PROFESSIONAL SCHOOL 1 7 (QO TO SX9C) 

REFUSED .7 (RECODE SX 8 = -7 AND 

GO TO BOX) 

DON’T KNOW .8 (RECODE SX 8 = -8 AND 

GO TO BOX) 



[if t: In this interview, we will be referring to that as "kindergarten." 
if p: In this interview, we will be referring to that as "prefirst grade."] 

SX9A. In terms of credits earned and requirements fulfilled, what year of vocational/ 
technical school [are you/is (person)] in now? 



XVOCYEAR 



FIRST 

SECOND OR HIGHER 2 

REFUSED .7 

DON’T KNOW -8 



(GOTOSX11) 

(GOTOSX11) 

(GOTOSX11) 

(GOTOSX11) 



What is [your/(PERSON’s)] class standing? That is, [are you/is (person)] a freshman, 
sophomore, junior, or senior? 



XCOLYEAR 



FRESHMAN 

SOPHOMORE 2 

JUNIOR 3 

SENIOR ; 4 

REFUSED .7 

DON’T KNOW -8 



(GOTOSX11) 

(GOTOSX11) 

(GOTOSX11) 

(GOTOSX11) 

(GOTOSX11) 

(GOTOSX11) 




EXPA 
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SX9C. 



In terms of credits earned and requirements fulfilled, what year of graduate 
or professional school [are you/is (person)] in now? 



XGRAYEAR 



RRST 1 (GO TO SX1 1) 

second:::::::: 2 (gotosxid 

THIRD ^ (GOTOSX11) 

FOURTH OR HIGHER 4 ( G0 T0 1 ) 

REFUSED - 7 (gotoSXII) 

DON'T KNOW ( G0 T0 ^*1 ^ ) 



SX10. 



SGRADEQ 



What grade would [you/(PERSON)] be in if [you/(he/she)] were (attending a school/ 
attending a school with regular grades)? 

[probe for t or p: Is that before or after kindergarten?] 

nursery/preschool/prekindergarten/head start N 

TRANSITIONAL KINDERGARTEN (BEFORE K) T 

KINDERGARTEN K 

PREFIRST GRADE (AFTER K) P 

FIRST GRADE 1 

SECOND GRADE 2 

THIRD GRADE 3 

FOURTH GRADE 4 

FIFTH GRADE 5 

SIXTH GRADE 6 

SEVENTH GRADE 7 

EIGHTH GRADE 8 

NINTH GRADE/FRESHMAN IN HIGH SCHOOL 9 

TENTH GRADE/SOPHOMORE IN HIGH SCHOOL 10 

ELEVENTH GRADE/JUNIOR IN HIGH SCHOOL 1 1 

TWELFTH GRADE/SENIOR IN HIGH SCHOOL 12 

UNGRADED/NO EQUIVALENT U 

VOCATIONAL/TECHNICAL AFTER HIGH SCHOOL 15 

COLLEGE (UNDERGRADUATE) 16 

GRADUATE, PROFESSIONAL SCHOOL 17 

REFUSED -7 . ( REC0DE SX8 = ' 7 AND 

GO TO BOX) 

DON’T KNOW - 8 (RECODE SX8 = -8 AND 

GO TO BOX) 

[if t: In this interview, we will be referring to that as "kindergarten." 
if p: In this interview, we will be referring to that as "prefirst grade."] 



If SX8 = 1 (in home school), go to first box after SX12. 
Else, ask SX11. 
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SX11. 



[Do you/Does (person)] go to a public or a private school? 



XPUBL PUBLIC 1 

PRIVATE 2 

REFUSED -7 

DON'T KNOW -8 



lfSX9 orSXIO = N, T, orKorSX7=1 and AGE >=16 or -7, -8 
(person enrolled in nursery school or kindergarten or age 16 or 
older and currently enrolled in school), then ask SX12. Else, go 
to first box after SX12. 



SX1 2. [Are you/ls (person)] now enrolled in school full time or part time? 



XFULTIM 



FULLTIME 1 

PART TIME 2 

REFUSED -7 

DON'T KNOW -8 



EXPA 



Ask SX7 to SX12 for next person enrolled in school. 
After last person, go to next box. 



If (SX7 and SX8 = 2, -7, -8) or (SX9 or SX10 = 15, 16, 17, -7, - 
8, and AGE >= 16 or -7, -8) (person age 16 and older who is not 
currently enrolled in grade/equivalent 12 or below, ungraded 
elementary or secondary, or special education), then askSX13 
to SX15. Else, go to first box after SX15. 
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SX13. 



[Now I have a few questions about (you/you and the other adults(s) in your household).] 
What is the highest grade or year of school that [you/(ADULT)] completed? 



XGRADE 

XGRADl 

XGRAD2 



UP TO 8TH GRADE 1 

9th TO 11TH GRADE 2 

1 2th grade but no diploma 3 

HIGH SCHOOL DIPLOMA/EQUIVALENT 4 

vocAtech program after high school but no 

VOC/TECH DIPLOMA 5 

vocAtech diploma after high school 6 

SOME COLLEGE BUT NO DEGREE 7 

ASSOCIATE'S DEGREE 8 

BACHELOR’S DEGREE 9 

GRADUATE OR PROFESSIONAL SCHOOL BUT NO DEGREE 10 

MASTER'S DEGREE (MA, MS) 11 

DOCTORATE DEGREE (PHD, EDD) 12 

PROFESSIONAL DEGREE AFTER BACHELOR'S DEGREE 

(medicine/md; dentistryAdds; law/jdAllb; etc.) 13 

REFUSED -7 

DON'T KNOW -8 



(ENTER ACTUAL GRADE, 

GO TO SX14) 

(ENTER ACTUAL GRADE, 

GOTO SX14) 

(GO TO SX14) 

(GO TO SX1 5) 

(GO TO SX14) 

(GO TO SX14) 

(GO TO SX14) 

(GO TO BOX AFTER SX14) 

(GO TO BOX AFTER SX14) 

(GO TO BOX AFTER SX14) 

(GO TO BOX AFTER SX1 4) 

(GO TO BOX AFTER SX1 4) 

(GO TO BOX AFTER SX14) 

(GO TO SX14) 

(GO TO SX14) 

ENUM 



SX14. [Do you/Does (adult)] have a high school diploma or its equivalent, 
such as a GED? 



1 
2 

7 

8 

EXPA 

Ask SX13 to SX14 for next person age 16 and older who is not 
currently enrolled in grade 12 or below, ungraded elementary or 
secondary, or special education. After last person, go to next 

box. 



SDIPLOMA 

YES 

NO 

REFUSED 

DON'T KNOW. 



If AGE >16 or -7, -8 (person age 16 or older), then ask SX1 5. 
After last person, go to SX16. 




31 



SX15. 



What is [your/(ADULrs)] marital status? [verify if known.] 



XMARSTAT 

MARRIED/REMARRIED 1 

SEPARATED 2 

DIVORCED 3 

WIDOWED 4 

NEVER MARRIED 5 

REFUSED -7 

DON'T KNOW -8 



XSCR 



SX16. Not counting the Reserves or National Guard, (are you/is any member of 

your household) currently serving on active duty in the U.S. Armed Forces? 



XHHACTV 



YES 1 (GO TO BOX) 

NO 2 (GO TO SX1 7) 

REFUSED .7 (GO TO SX1 7) 

DON'T KNOW . 8 (GO TO SX17) 



_ ==== ^ EXPA 

If SX1 6 = 1 (on active duty) and respondent is the only adult in the 
household, autocode SX160V to respondent's person number. 

Else, go to SX160V. 



SX1 60V. (Who is that?) [display household members with age >= 1 6. code all 

THAT APPLY. IF RESPONDENT IS THE ONLY HOUSEHOLD MEMBER, CODE THE PERSON 
NUMBER OF THE RESPONDENT.] 



XACTVDUT 

PERSON NUMBER 

REFUSED 

DON’T KNOW 



□ □ (INELIGIBLE) 

-7 

-8 



XSCR 



SX1 7. (Were you/Was everyone in your household) born in this country, that is, 

in one of the 50 States or the District of Columbia? 



XHHBORN 



YES 1 

NO 2 

REFUSED ! .7 

DON’T KNOW -8 



(GO TOSX18) 

(GO TO BOX AFTER SX18) 
(GO TO BOX AFTER SX1 8) 
(GO TO BOX AFTER SX18) 
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SX18. 



(Did you/Did every member of your household) learn English as (your/their) 
first language? 



XHHLANG 



YES 

NO 

REFUSED 

DON’T KNOW. 



(GOTO 1ST BOX AFTER 

SX20) 

(GO TO BOX) 

(GO TO BOX) 

(GO TO BOX) 



If SX1 7 = 2,-7, -8 (not every household member was bom in the 
U.S.), then ask SX19 for each person in the household; also ask 
SX20 for each person where AGE >= 3, -7, -8 ( age 3 or older). 

If SX1 8 = 2, -7, -8 (not every household member learned 
English as their first language), ask SX20 for each person where 
I AGE >= 3 or -7, -8 (age 3 or older). | 

EXPA 

SX1 9. In what country [were you/was (person)] born? 

XBORNUS 

50 STATES OR THE DISTRICT OF COLUMBIA 1 

u.s. territories: Puerto rico, Guam, American Samoa, 

MARIANA ISLANDS, U.S. VIRGIN ISLANDS, OR SOLOMON ISLANDS 2 

XBORNOSl (SPECIFY) 

SOME OTHER COUNTRY 3 

XBORNOS2 (SPECIFY) 

REFUSED ’ 7 

DON’T KNOW * 8 

SX20. What was the first language [you/(PERSON)] learned to speak? 



XLANG ENGLISH ' 

SPANISH - 2 

SPANISH AND ENGLISH EQUALLY 3 

OTHER LANGUAGE 91 

XLANGOS (SPECIFY) 

REFUSED -7 

DON’T KNOW -8 



Ask SX21 and SX22 for each person. After last person, go to 
first box after SX22 (Sampling Point). 
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SX21. 



[Are you/ls (person)] .... 

[IF R GIVES RACE AND ALSO SAYS HE/SHE IS OF HISPANIC ORIGIN, CODE RACE HERE.] 



EXPA.RACE 

White 1 

Black 2 

American Indian or Alaskan Native 3 

Asian or Pacific Islander, or 4 

Some other race? 5 

REFUSED -7 

DON'T KNOW -8 



SX21A. [CODE RESPONSE IF SX21 = 5.] 



(GO TO SX22) 
(GO TO SX22) 
(GO TO SX22) 
(GO TO SX22) 
(GOTO SX21 A) 
(GOTO SX21) 
(GOTO SX21) 



EXPA.OTH 

HISPANIC/LATINO/MEXICAN/SPANISH/ 

PUERTO RICAN 1 (AUTOCODE SX22=1 AND 

GO TO BOX AFTER SX22) 

MORE THAN ONE RACE/BIRACIAL/MULTIRACIAL 2 

OTHER 91 

(SPECIFY) ' 



SX22. [Are you/Is (person)] of Hispanic origin? 



EXPA.HISPANC 

YES 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 



Sampling Point: 

Select children and adults for extended interviews. 

If any children are selected, go to next box. 

If adults only are selected, go to second box after SX24. 

If no one is selected, go to UNTRO. 

Ask SX23 and SX24 for each sampled child. If there is only one 
household member, other than the sampled child, who is at 
least 12 years older than that child, autocode SX23 to this adult. 

If the sampled child is the only or oldest person in the 
household, go to THANK2. 

ENUM 
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SX23. We would like to ask some questions about (your/(CHiLD)'s) 

(care and) education. 

RESPNUM [IF SCREENER RESPONDENT IS OBVIOUSLY CHILD'S MOTHER, ASK: 

Are you (child)'s mother? if yes, enter her person number.] 

[IF SCREENER RESPONDENT MIGHT NOT BE CHILD'S MOTHER, ASK: 

Who is the parent or guardian in this household who knows 
the most about (your/(CHiLD)’s) (care and) education?] 

[DISPLAY HOUSEHOLD MEMBERS 1 6 YEARS OLD AND OLDER.] 

PERSON NUMBER □□ 



If person number at SX23 NE person number of sampled child, 
ask SX24 for each sampled child, and then go to HHSELECT 
screen to select interview. If person number at SX23 = person 
number of sampled child, display household members at least 
12 years older than the sampled child and ask SX24 for each 
household member listed. The respondent for the parent 
interview will be selected in the following order of relationship: 
mother, father, grandmother, aunt, sister, grandfather, uncle, 
brother, cousin, other relative (but not husband/wife), or 
nonrelative (but not boyfriend/girlfriend). If no household 
member is so designated, the sampled child is ineligible; go to 
LINTRO. If a respondent for the parent interview is selected, go 
to HHSELECT screen to select interview. 



SX24. What is [your/(CAREGiVER's)/(PERSON's)] relationship to [(CHiLD)/you]? 

[VERIFY IF KNOWN] 



PARRELN MOTHER (BIRTH/ADOPTIVE/STEP/FOSTER) 1 

FATHER (BIRTH/ADOPTIVE/STEP/FOSTER) 2 

BROTHERS, INCLUDING STEP, 

ADOPTED, AND FOSTER 3 

SISTERS, INCLUDING STEP, 

ADOPTED, AND FOSTER 4 

GRANDPARENT 5 

AUNT 6 

UNCLE 7 

COUSIN 8 

OTHER RELATIVE/GUARDIAN (BUT NOT HUSBAND/WIFE) 9 

NONRELATIVE/GUARDIAN (BUT NOT BOYFRIEND/GIRLFRIEND)... 10 

HUSBAND/WIFE/BOYFRIEND/GIRLFRIEND 1 1 

REFUSED -7 

DON’T KNOW -8 
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If PARRELN = 5 (grandparent) use that person's sex with 
PARRELN to set RELATION and drive displays. 

If sampled adult is not the screener respondent and SX7 = 1 
and SX9 =15, 16, 17 (enrolled in college, graduate school or 
vocationalAechnical school after high school) for that person, 
ask SX25. Else, go to HHSELECT (Adult Cl extended 

interview). 



SX25. Is (adult) living there, in student housing, or somewhere else? 



UVENOW 

HERE 1 

student housing [This includes all housing owned, 
sponsored, or leased by the school such as a 

dormitory or fraternity or sorority house.) 2 

OTHER PRIVATE HOME OR APARTMENT 3 

INSTITUTION OR GROUP QUARTERS [THIS INCLUDES 
A JAIL OR DETENTION CENTER, MEDICAL FACILITY, 
REHABILITATION CENTER, MENTAL HEALTH FACILITY, 

MILITARY BARRACKS, OR GROUP FOSTER CARE.) 4 

REFUSED -7 

DON’T KNOW -8 



SX26. Would you please give me (his/her) last name and telephone number 
so that we can call (him/her) to do a brief interview about activities 
related to civic involvement? 



(GO TO HHSELECT) 

(GO TO SX26) 

(INELIGIBLE, GO TO THANK2) 



(INELIGIBLE, GO TO THANK2) 
(GO TO HHSELECT) 

(GO TO HHSELECT) 



LNAME LAST NAME 

ENUM.ENUMAREA PHONE 

EXCH REFUSED -7 

LOCL DON'T KNOW -8 



1996 Topical Component: Household Public Library Usage 

LINTRO. (These next questions are about/We are interested in) public libraries. 

This does not include school or college libraries, or special research 
libraries. 



LI • About how far would you say it is from your home to the closest public library? 

Would you say... 



LDISTANC 

Less than 1 mile 1 

1 or 2 miles, 2 

3 to 5 miles 3 

6 to 10 miles, or 4 

More than 10 miles? 5 

REFUSED -7 

DON’T KNOW -8 



O 

ERLC 
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I 2 People use public libraries in a number of ways. In the past month , that is, 

since (month) (day), has any member of your household used a public library 
in the following ways? How about... 









YES 


NO 


R 


DK 


LVISITl 


a. 


Going to a public library to borrow or drop off 

books or tapes? 


1 


2 


-7 


-8 


LVISIT2 


b. 


Going to a public library for any other purpose, such as 
a lecture or story hour, or to use their equipment? 


1 


2 


-7 


-8 


LCOMP 


c. 


Using a home computer to link to a public library? 


1 


2 


-7 


-8 


LPHONE 


d. 


Calling the public library to renew books or for 
information other than library hours or directions? 


1 


2 


-7 


-8 


LMATLS 


e. 


Having library materials mailed or delivered to your 

hnmp? 


1 


2 


-7 


-8 


LMOBILE 


f. 


Visiting a bookmobile? 


1 


2 


-7 


-8 



If L2a through L2f = 2 (no use of public libraries), ask L3. 
Else, go to L4. 



L3. Has anyone in your household used a public library in any of those ways 

in the past year ? 



(GO TO BOX AFTER L4) 
(GO TO BOX AFTER L4) 
(GO TO BOX AFTER L4) 
(GO TO BOX AFTER L4) 

L4. In the past month, that is since (month) (day), has any member of your 

household used a public library (or bookmobile) for the following purposes? 

How about... 









YES 


NO 


R DK 


LSCHOOL 


a. 


For a school or class assignment? 


1 


2 


-7 -8 


LKIDSACT 


b. 


A program or activity designed for children age 6 to 12? 


1 


2 


-7 -8 


LKIDBOOK 


c. 


An activity for children under 6, such as story hour 












or other introduction to books and reading? 


1 


2 


-7 -8 


LRECR 


d. 


For enjoyment or hobbies, including to borrow 












books or tapes or attend activities? 


1 


2 


-7 -8 


LJOBHELP 


e. 


To get information to help find a job? 


1 


2 


-7 -8 


LWORK 


f. 


For a work assignment or to keep up to date 












at work? 


1 


2 


-7 -8 


LCONSUME 


g- 


To get information for personal use, such as 










consumer or health issues, investments, and so on? 


1 


2 


-7 -8 


LLRNREAD 


h. 


To work with a tutor or take a class to learn 












to read? 


1 


2 


-7 -8 



LYRUSE 

YES 1 

NO 2 

REFUSED -7 

DON’T KNOW -8 



o 

FRir 
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If library items are administered before matrix, go to PRE_S6. If 
library items are administered in the Parent Interview, go to 

SX27. 



Household Characteristics 



XSCR 



SX27. Now a few more questions about your household. Do you... 



hownhome Own your home 1 

Rent your home, or 2 

Have some other arrangement? 3 

refused -7 

DON'T KNOW -8 



SX28. Besides (phone number), do you have other telephone numbers 

in your household? 



HOTHNUM 

YES 1 (GO TO SX29) 

NO 2 (GO TO SX30) 

NOT MY NUMBER [RECORD NUMBER IN COMMENTS] 3 (GO TO SX30) 

REFUSED -7 (GO TO SX30) 

DON'T KNOW -8 (GO TO SX30) 



SX29. How many of these additional telephone numbers are for home use? 



HNUMUSE 

NUMBER □□ 

REFUSED -7 

DON'T KNOW -8 



SX30. So that we can group households geographically, may I have 

your ZIP code? 



HZIPCODE 

ZIP CODE □□□□□ 

REFUSED -7 

DON'T KNOW -8 

SX31 . Which of these best describes the community where you live? Is it... 



HCCOMMUN 

A rural or farming community 1 

A suburb of a city 2 

A small city or town of fewer than 

50,000 people, or 3 

A city of 50,000 people or more? 4 

refused -7 

DON'T KNOW -8 




4 



2 
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If SX31 = 2 (suburb), go to SX310V. Else, if SX31 = 4 (city of 
50,000 people or more), go to SX310V2. 

Else, go to box after SX310V2. 



SX31 OV. Is it a suburb of... 



HCSUB 

A city with over 500,000 people, 1 

A city with 1 00,000 to 500,000, or 2 

A city with 50,000 to 1 00,000? 3 

REFUSED " 7 

DON’T KNOW ' 8 

SX310V2. Is it ... 



(GO TO BOX AFTER SX31 OV2) 
(GO TO BOX AFTER SX31 OV2) 
(GO TO BOX AFTER SX31 OV2) 
(GO TO BOX AFTER SX31 OV2) 
(GO TO BOX AFTER SX31 OV2) 



HCCITY 

A city with over 500,000 people 1 

A city with 1 00,000 to 500,000, or 2 

A city with 50,000 to 100,000? 3 

REFUSED " 7 

DON'T KNOW -8 



Ask SX32 if NUMKID20 (number of children age 20 or 
younger) >= 1. Else, go to SX33. 



SX32. In the past 12 months , has your family received funds or services 

from any of the following programs? How about... 







YES 


NO 


R 


DK 


HWIC 


a. Women, Infants, and Children, or WIC? 


1 


2 


-7 


-8 


HFOODST 


b. Food Stamps? "• 


1 


2 


-7 


-8 


HAFDC 


c. AFDC, or Aid to Families with Dependent 








-8 




Children? 


1 


2 


-7 


SX33. 


In studies like this, households are sometimes grouped according to income. 







What was the total income of all persons in your household over the past year, 
including salaries or other earnings, interest, retirement, and so on for all 
household members? 
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d3 



HINCMRNG 



Was it... 



HINCOME 



$25,000 or less, or.. 
More than $25,000? 

REFUSED 

DON'T KNOW 

Was it... 



(READ SET 1) 
(READ SET 2) 

(GO TO THANK2) 
(GO TO THANK2) 



[SET 1] 

$5,000 or less 

$5,001 to $10, 000 

$10,001 to $15, 000 

$15,001 to $20,000, or 
$20,001 to $25,000?... 



[SET 2] 

$25,001 to $30, 000 

$30,001 to $35,000 

$35,001 to $40, 000 

$40,001 to $50, 000 

$50,001 to $75,000, or 

Over $75,000? 

REFUSED 

DON'T KNOW 



(GO TO THANK2) 
(GO TO THANK2) 



Ask SX330V if 

(Number in HH = 2 and HINCOME < = 2) or 
(Number in HH = 3 and HINCOME < = 3) or 
(Number in HH = 4 and HINCOME < = 3) or 
(Number in HH = 5 and HINCOME < = 4) or 
(Number in HH = 6 and HINCOME < = 4) or 
(Number in HH = 7 and HINCOME < = 5) or 
(Number in HH = 8 and HINCOME < = 5) or 
(Number in HH = 9 and HINCOME < = 6) or 
(Number in HH = 10 and HINCOME < = 6) or 
(Number in HH = 11 and HINCOME < = 7)or 
(Number in HH = 12 and HINCOME < = 7). 
Else, go to THANK2. 



SX330V. 

HINCMEXT 



THANK1. 

THANK2. 



What was your total income last year, to the nearest thousand? 



amount 

REFUSED 

DON’T KNOW 

Thank you, but we are only 

Those are all the questions 





-7 



interviewing in private residences. 

I have about your household. Thank you for your time. 



Appendix B: 



Advance Letter 
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OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 



U.S. DEPARTMENT OF EDUCATION 



NATIONAL CENTER FOR EDUCATION STATISTICS 



August, 1995 



Dear Potential Study Participant, 

I am writing to strongly encourage your participation in an upcoming telephone survey, the 
National Household Education Survey (NHES). One purpose of the NHES is to find out how families and 
schools work together to shape the learning experiences of children. We are also interested in how both 
young people and adults learn about and are involved in their communities and their country. Finally, we 
want to know about how people use public libraries. Your household may be asked about one or more of 
these topics. 

The NHES is sponsored by the National Center for Education Statistics of the United States 
Department of Education. Your telephone number was selected for the study as part of a scientific 
random sample of all households in the nation, and another number cannot be substituted for yours. Your 
household represents thousands of households in the United States. It is very important that you take part 
in our survey so that our results show a true picture for the whole nation. 

Westat, Inc., a professional research firm, will be conducting the NHES. The study will take 
place from January 2 to March 31. Sometime during that time, a Westat interviewer will call you. If we 
happen to call at an inconvenient time, please suggest a time that is better for you. If you would like to set 
an appointment anytime before we call, contact Westat at their toll-free number (1-800-862-9452). You 
will need to give your phone number and your preferred appointment time. 

Some questions and answers about the NHES are printed on the back of this letter. I hope you 
will take part in this very important study. 



Sincerely, 




Kathryn Chandler 
Project Officer 

National Household Education Survey 
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Some Frequently Asked Questions about 
the National Household Education Survey (NHES) 



Q. How will the study results be used? What will you do with this information? 

A. The information we collect will tell us about people's experiences with schools, libraries, and 

their communities, and help us understand educational experiences and needs. Some information 
from the study will be published in the annual report on the National Education Goals. Other 
findings will be published in U.S. Department of Education reports. Those reports will be widely 
distributed to educators, researchers, news organizations, and the general public. 

Q. How did you get my telephone number? 

A. Your telephone number was randomly selected from all of the possible telephone numbers in the 
nation. We do not use telephone directories to select telephone numbers. 

Q. How did you get my address? 

A. After the telephone numbers were randomly selected, we sent them to a company which provided 
addresses for those telephone numbers. That company gave us the file that was used to address 
the envelopes for this mailing. No records were kept of the addresses after this mailing was 
completed, and addresses are not on the file that contains the telephone numbers. Interviewers do 
not have the names or addresses for any telephone numbers. 

Q. Will you keep my information confidential? 

A. All information you give to the interviewer will be kept completely confidential. All employees 
of the U.S. Department of Education and Westat who are working on this study are required by 
law to protect the confidentiality of respondents. Also, individual responses are never published 
in reports; they are added to the responses of others and are published as combined information 
only. 

Q. How long will the survey take? 

A. All households are asked to participate in an initial interview that usually takes 5 to 7 minutes. 
This includes questions about household members and about public library use. We use 
information about household members to determine whether anyone in your household will be 
selected for an additional interview. In more than half of the households, no one is selected for an 
additional interview. In other households, we ask questions about how families and schools work 
together to shape children's learning experiences and/or how young people and adults learn about 
and are involved in their communities and their country. These additional interviews take about 
15 to 20 minutes. 
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